Detecting goals of care conversations in clinical notes with active learning.

OBJECTIVE: Goals of care (GOC) discussions are an increasingly used quality metric in serious illness care and research. Wide variation in documentation practices within the Electronic Health Record (EHR) presents challenges for reliable measurement of GOC discussions. Novel natural language processing approaches are needed to capture GOC discussions documented in real-world samples of seriously ill hospitalized patients' EHR notes, a corpus with a very low event prevalence. METHODS: To automatically detect sentences documenting GOC discussions outside of dedicated GOC note types, we proposed an ensemble of classifiers aggregating the predictions of rule-based, feature-based, and three transformers-based classifiers. We trained our classifier on 600 manually annotated EHR notes among patients with serious illnesses. Our corpus exhibited an extremely imbalanced ratio between sentences discussing GOC and sentences that do not. This ratio challenges standard supervision methods to train a classifier. Therefore, we trained our classifier with active learning. RESULTS: Using active learning, we reduced the annotation cost to fine-tune our ensemble by 70% while improving its performance in our test set of 176 EHR notes, with 0.557 F1-score for sentence classification and 0.629 for note classification. CONCLUSION: When classifying notes, with a true positive rate of 72% (13/18) and false positive rate of 8% (13/158), our performance may be sufficient for deploying our classifier in the EHR to facilitate bedside clinicians' access to GOC conversations documented outside of dedicated notes types, without overburdening clinicians with false positives. Improvements are needed before using it to enrich trial populations or as an outcome measure.

Association of Unit Census with Delays in Antimicrobial Initiation among Ward Patients with Hospital-acquired Sepsis.

Rationale: Patients with hospital-acquired sepsis (HAS) experience higher mortality and delayed care compared with those with community-acquired sepsis. Capacity strain, the extent to which demand for hospital resources exceeds availability, thus impacting patient care, is a possible mechanism underlying antimicrobial delays for HAS but has not been studied. Objectives: Assess the association of ward census with the timing of antimicrobial initiation among ward patients with HAS. Methods: This retrospective cohort study included adult patients hospitalized at five acute care hospitals between July 2017 and December 2019 who developed ward-onset HAS, distinguished from community-acquired sepsis by onset after 48 hours of hospitalization. The primary exposure was ward census, measured as the number of patients present in each ward at each hour, standardized by quarter and year. The primary outcome was time from sepsis onset to antimicrobial initiation. We used quantile regression to assess the association between ward census at sepsis onset and time to antimicrobial initiation among patients with HAS defined by Centers for Disease Control and Prevention Adult Sepsis Event criteria. We adjusted for hospital, year, quarter, age, sex, race, ethnicity, severity of illness, admission diagnosis, and service type. Results: A total of 1,672 hospitalizations included at least one ward-onset HAS episode. Median time to antimicrobial initiation after HAS onset was 4.1 hours (interquartile range, 0.4-22.3). Marginal adjusted time to antimicrobial initiation ranged from 3.6 hours (95% confidence interval [CI], 2.4-4.8 h) to 6.8 hours (95% CI, 5.3-8.4 h) at census levels 2 standard deviations (SDs) below and above the ward-specific mean, respectively. Each 1-SD increase in ward census at sepsis onset, representing a median of 2.4 patients, was associated with an increase in time to antimicrobial initiation of 0.80 hours (95% CI, 0.32-1.29 h). In sensitivity analyses, results were consistent across severity of illness and electronic health record-based sepsis definitions. Conclusions: Time to antimicrobial initiation increased with increasing census among ward patients with HAS. These findings suggest that delays in care for HAS may be related to ward capacity strain as measured by census. Additional work is needed to validate these findings and identify potential mechanisms operating through clinician behavior and care delivery processes.

Word embeddings trained on published case reports are lightweight, effective for clinical tasks, and free of protected health information.

OBJECTIVE: Quantify tradeoffs in performance, reproducibility, and resource demands across several strategies for developing clinically relevant word embeddings. MATERIALS AND METHODS: We trained separate embeddings on all full-text manuscripts in the Pubmed Central (PMC) Open Access subset, case reports therein, the English Wikipedia corpus, the Medical Information Mart for Intensive Care (MIMIC) III dataset, and all notes in the University of Pennsylvania Health System (UPHS) electronic health record. We tested embeddings in six clinically relevant tasks including mortality prediction and de-identification, and assessed performance using the scaled Brier score (SBS) and the proportion of notes successfully de-identified, respectively. RESULTS: Embeddings from UPHS notes best predicted mortality (SBS 0.30, 95% CI 0.15 to 0.45) while Wikipedia embeddings performed worst (SBS 0.12, 95% CI -0.05 to 0.28). Wikipedia embeddings most consistently (78% of notes) and the full PMC corpus embeddings least consistently (48%) de-identified notes. Across all six tasks, the full PMC corpus demonstrated the most consistent performance, and the Wikipedia corpus the least. Corpus size ranged from 49 million tokens (PMC case reports) to 10 billion (UPHS). DISCUSSION: Embeddings trained on published case reports performed as least as well as embeddings trained on other corpora in most tasks, and clinical corpora consistently outperformed non-clinical corpora. No single corpus produced a strictly dominant set of embeddings across all tasks and so the optimal training corpus depends on intended use. CONCLUSION: Embeddings trained on published case reports performed comparably on most clinical tasks to embeddings trained on larger corpora. Open access corpora allow training of clinically relevant, effective, and reproducible embeddings.

Development and validation of a prediction model for actionable aspects of frailty in the text of clinicians' encounter notes.

OBJECTIVE: Frailty is a prevalent risk factor for adverse outcomes among patients with chronic lung disease. However, identifying frail patients who may benefit from interventions is challenging using standard data sources. We therefore sought to identify phrases in clinical notes in the electronic health record (EHR) that describe actionable frailty syndromes. MATERIALS AND METHODS: We used an active learning strategy to select notes from the EHR and annotated each sentence for 4 actionable aspects of frailty: respiratory impairment, musculoskeletal problems, fall risk, and nutritional deficiencies. We compared the performance of regression, tree-based, and neural network models to predict the labels for each sentence. We evaluated performance with the scaled Brier score (SBS), where 1 is perfect and 0 is uninformative, and the positive predictive value (PPV). RESULTS: We manually annotated 155 952 sentences from 326 patients. Elastic net regression had the best performance across all 4 frailty aspects (SBS 0.52, 95% confidence interval [CI] 0.49-0.54) followed by random forests (SBS 0.49, 95% CI 0.47-0.51), and multi-task neural networks (SBS 0.39, 95% CI 0.37-0.42). For the elastic net model, the PPV for identifying the presence of respiratory impairment was 54.8% (95% CI 53.3%-56.6%) at a sensitivity of 80%. DISCUSSION: Classification models using EHR notes can effectively identify actionable aspects of frailty among patients living with chronic lung disease. Regression performed better than random forest and neural network models. CONCLUSIONS: NLP-based models offer promising support to population health management programs that seek to identify and refer community-dwelling patients with frailty for evidence-based interventions.

Locally Informed Simulation to Predict Hospital Capacity Needs During the COVID-19 Pandemic.

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic challenges hospital leaders to make time-sensitive, critical decisions about clinical operations and resource allocations. OBJECTIVE: To estimate the timing of surges in clinical demand and the best- and worst-case scenarios of local COVID-19-induced strain on hospital capacity, and thus inform clinical operations and staffing demands and identify when hospital capacity would be saturated. DESIGN: Monte Carlo simulation instantiation of a susceptible, infected, removed (SIR) model with a 1-day cycle. SETTING: 3 hospitals in an academic health system. PATIENTS: All people living in the greater Philadelphia region. MEASUREMENTS: The COVID-19 Hospital Impact Model (CHIME) (http://penn-chime.phl.io) SIR model was used to estimate the time from 23 March 2020 until hospital capacity would probably be exceeded, and the intensity of the surge, including for intensive care unit (ICU) beds and ventilators. RESULTS: Using patients with COVID-19 alone, CHIME estimated that it would be 31 to 53 days before demand exceeds existing hospital capacity. In best- and worst-case scenarios of surges in the number of patients with COVID-19, the needed total capacity for hospital beds would reach 3131 to 12 650 across the 3 hospitals, including 338 to 1608 ICU beds and 118 to 599 ventilators. LIMITATIONS: Model parameters were taken directly or derived from published data across heterogeneous populations and practice environments and from the health system's historical data. CHIME does not incorporate more transition states to model infection severity, social networks to model transmission dynamics, or geographic information to account for spatial patterns of human interaction. CONCLUSION: Publicly available and designed for hospital operations leaders, this modeling tool can inform preparations for capacity strain during the early days of a pandemic. PRIMARY FUNDING SOURCE: University of Pennsylvania Health System and the Palliative and Advanced Illness Research Center.